Electronic device and method of operating the same

ABSTRACT

Provided is an electronic device for detecting lost portions of a field of vision and distorted portions of a field of vision and storing these in a vision map. The lost portions and distorted portions are identified in cooperation with the user by means of virtual image superposition on a real scene, audible requests to the user, gesture inputs provided by the user and gaze tracking. The electronic device may detect an object and provide visual information to a user when the object is in a lost portion or distorted portion of the field of vision.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a by-pass continuation of international applicationPCT/KR2022/002508 filed on Feb. 21, 2022 and claims benefit of priorityto Korean Patent Application No. 10-2021-0024366 filed on Feb. 23, 2021.

TECHNICAL FIELD

The disclosure relates to an electronic device for diagnosing visualimpairment of a user and providing assistance to the user with visualimpairment while providing an augmented reality (AR) service.

BACKGROUND ART

Augmented reality (AR) is a technology for overlaying a virtual image ona physical environment space of the real world or on a real worldobject, so as to be displayed as a single image. An AR device which isworn on the face or head of a user allows the user to see a real sceneand a virtual image together through a see-through display moduleprovided in front of the eyes of the user.

Research on a method of easily and accurately diagnosing visualimpairment of the user by using the AR device when the user of the ARdevice has visual impairment is required.

In addition, research on a method of minimizing inconvenience caused byvisual impairment when providing an AR service to the user with visualimpairment is also required.

DESCRIPTION OF EMBODIMENTS Technical Problem

Provided are an electronic device for diagnosing visual impairment of auser, and a method of operating the same.

Provided are an electronic device for providing assistance to a userwith visual impairment, and a method of operating the same.

Technical problems to be solved are not limited to the above-describedtechnical problems, and other technical problems may also be present.

Technical Solution to Problem

Provided herein is an electronic device including: a display includingan optical engine and a waveguide; a gaze tracking sensor; a memoryconfigured to store one or more instructions; and a processor configuredto execute the one or more instructions to: control the optical engineto output a first target image through the waveguide at preset differentlocations; obtain gaze information of eyes of a user corresponding to alocation of the first target image by using the gaze tracking sensor;determine whether a gaze of the eyes of the user is directed to thelocation where the first target image is output, based on the gazeinformation; according to a first result of the determination, determinean impaired area of an entire visual field; and store a vision map basedon the impaired area.

Also provided herein is a method of operating an electronic device, themethod including: controlling an optical engine to output a first targetimage through a waveguide at preset different locations; obtaining gazeinformation of eyes of a user corresponding to a location of the firsttarget image by using a gaze tracking sensor when the first target imageis output; determining, based on the gaze information, whether a gaze ofthe eyes of the user is directed to the first target image; according toa first result of the determination, determining an impaired area of anentire visual field; and storing a vision map based on the impairedarea.

According to another embodiment of the disclosure, a computer-readablerecording medium has recorded thereon a computer program for executingthe above-described method.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a view for describing an existing visual impairment diagnosismethod.

FIG. 1B is a view for describing an embodiment of the disclosure.

FIG. 1C is a view for describing a visual impairment diagnosis methodaccording to an embodiment of the disclosure.

FIG. 2 is a block diagram of an electronic device according to anembodiment of the disclosure.

FIG. 3A is a view showing an example of an electronic device accordingto an embodiment of the disclosure.

FIG. 3B is a view for describing an optical engine and a waveguide,according to an embodiment of the disclosure.

FIG. 4A is a view for describing a method of operating a gaze trackingsensor, according to an embodiment of the disclosure.

FIG. 4B is a view for describing a method of operating a gaze trackingsensor, according to another embodiment of the disclosure.

FIG. 5 is a view showing an example in which refractive power of avarifocal lens is changed, according to an embodiment of the disclosure.

FIG. 6 is a view for describing a prism mode of a varifocal lens,according to an embodiment of the disclosure.

FIG. 7 is a flowchart of a method, performed by an electronic device, ofdiagnosing visual impairment based on obtaining of gaze information,according to an embodiment of the disclosure.

FIG. 8 is a view for describing an example in which an electronic devicediagnoses visual impairment based on obtaining of gaze information,according to an embodiment of the disclosure.

FIG. 9 is a view for describing an example in which an electronic devicediagnoses visual impairment of the left or right eye, according to anembodiment of the disclosure.

FIG. 10 is a view for describing another example in which an electronicdevice diagnoses visual impairment of the left or right eye, accordingto another embodiment of the disclosure.

FIG. 11 is a flowchart of a method, performed by an electronic device,of diagnosing visual impairment based on obtaining of a gesture input,according to an embodiment of the disclosure.

FIG. 12 is a view for describing an example in which an electronicdevice diagnoses visual impairment based on obtaining of a gestureinput, according to an embodiment of the disclosure.

FIG. 13 is a view for describing another example in which an electronicdevice diagnoses visual impairment based on obtaining of a gestureinput, according to an embodiment of the disclosure.

FIG. 14 is a view for describing an example in which an electronicdevice detects a distorted area, according to an embodiment of thedisclosure.

FIG. 15 is a view for describing another example in which an electronicdevice detects a distorted area, according to an embodiment of thedisclosure.

FIG. 16 is a view for describing another example in which an electronicdevice detects an impaired area of vision of a user, according to anembodiment of the disclosure.

FIG. 17 is a flowchart of a method, performed by an electronic device,of calculating a degree of distortion, according to an embodiment of thedisclosure.

FIG. 18 is a view for describing an example in which an electronicdevice calculates a degree of distortion, according to an embodiment ofthe disclosure.

FIG. 19 is a flowchart of a method, performed by an electronic device,of outputting a guidance image for notifying an impaired area of visionof a user, according to an embodiment of the disclosure.

FIG. 20 is a view for describing an example in which an electronicdevice outputs a guidance image for notifying an impaired area,according to an embodiment of the disclosure.

FIG. 21 is a view for describing an example in which an electronicdevice outputs a guidance image on a display, according to an embodimentof the disclosure.

FIG. 22 is a flowchart of a method, performed by an electronic device,of outputting an object based on a prism mode, according to anembodiment of the disclosure.

FIG. 23 is a view for describing application of a prism mode, accordingto an embodiment of the disclosure.

FIG. 24 is a view for describing an example in which an electronicdevice outputs an object based on a prism mode, according to anembodiment of the disclosure.

MODE OF DISCLOSURE

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Hereinafter, the disclosure will be described in detail by explainingembodiments of the disclosure with reference to the attached drawings.The disclosure may, however, be embodied in many different forms andshould not be construed as being limited to the embodiments of thedisclosure set forth herein. In the drawings, parts not related to thedisclosure are not illustrated for clarity of explanation, and likereference numerals denote like elements throughout.

Although the terms used herein are selected, as much as possible, fromgeneral terms that are widely used at present while taking intoconsideration the functions obtained in accordance with the disclosure,these terms may be replaced by other terms based on intentions of one ofordinary skill in the art, customs, emergence of new technologies, orthe like. Therefore, it is noted that the terms used herein areconstrued based on practical meanings thereof and the whole content ofthis specification, rather than being simply construed based on names ofthe terms.

Terms such as “first” and “second” may be used to designate variouselements, but the elements should not be limited by these terms. Theseterms are merely used to distinguish one element from another.

Terms in the following description are merely used to describe specificembodiments of the disclosure, and are not intended to limit the scopeof the disclosure. The singular forms “a”, “an”, and “the” are intendedto include the plural forms as well, unless the context clearlyindicates otherwise. Throughout the specification, when an element isreferred to as being “connected to” another element, the element can be“directly connected to” the other element or be “electrically connectedto” the other element via an intervening element. The terms “comprises”,“comprising”, “includes” and/or “including”, when used herein, specifythe presence of stated elements, but do not preclude the presence oraddition of one or more other elements.

The definite article “the” or other demonstratives may indicate both asingular form and a plural form. Unless the context clearly indicatesotherwise, operations included in a method according to an embodiment ofthe disclosure may be performed in an appropriate order. The order ofdescribing the operations does not limit the scope of the disclosure.

The phrase “an embodiment of the disclosure” at various parts of thisspecification does not always designate the same embodiment of thedisclosure.

An embodiment of the disclosure may be represented as functional blocksand various processing steps. Some or all of the functional blocks maybe implemented by various numbers of hardware and/or software elementsconfigured to perform certain functions. For example, the functionalblocks of the disclosure may be implemented by one or moremicroprocessors or circuit elements for certain functions. As anotherexample, the functional blocks of the disclosure may be implementedusing various programming or scripting languages. The functional blocksmay be implemented using algorithms executed by one or more processors.Furthermore, the disclosure might employ known technologies forelectronic settings, signal processing, and/or data processing. Termssuch as “mechanism”, “element”, “means”, and “configuration” may bewidely used and are not limited to mechanical and physicalconfigurations.

In addition, connection lines or connection members between elementsshown in the drawings merely illustrate examples of functionalconnections and/or physical or circuit connections. Connections betweenelements may be represented by replaceable or additional variousfunctional connections, physical connections, or circuit connections inan actual device.

As used herein, ‘augmented reality (AR)’ refers to a technology fordisplaying a virtual image on a physical environment space of the realworld or displaying a real world object and a virtual image together.

An ‘AR device’ is a device capable of implementing ‘augmented reality’,and generally includes not only AR glasses which are worn on the face ofa user but also a head mounted display (HMD) or an AR helmet which isworn on the head.

A ‘real scene’ is a scene of the real world which is seen by the userthrough the AR device, and may include a real world object. A ‘virtualimage’ is an image formed by an optical engine and may include both astill image and a moving image. The virtual image is seen together withthe real scene, and may be an image including information about the realworld object in the real scene, information about operation of the ARdevice, a control menu, or the like.

Therefore, a general AR device includes an optical engine for forming avirtual image by using light generated by a light source, and awaveguide for guiding the virtual image formed by the optical engine, tothe eyes of the user, and made of a transparent material to allow ascene of the real world to be seen therethrough. As described above,because the AR device needs to allow the scene of the real world to beseen therethrough, an optical element for changing a path of light,which basically has linearity, is required to guide the light from theoptical engine through waveguide to the eyes of the user. In this case,the path of light may be changed using reflection by a mirror or usingdiffraction by a diffractive element such as a diffractive opticalelement (DOE) or a holographic optical element (HOE), but is not limitedthereto.

Reference will now be made in detail to embodiments of the disclosure,examples of which are illustrated in the accompanying drawings.

FIG. 1A is a view for describing an existing visual impairment diagnosismethod.

As illustrated in FIG. 1A, for example, an Amsler grid test may havebeen used for a user to self-diagnose visual impairment. When looking ata dot in the center of the grid with one eye closed, a first grid 51 maybe seen by a user with normal vision. However, a user with visualimpairment due to aging of the eyes or an eye disease such as maculardegeneration may see a wavy or blank area on the grid. For example, asecond grid 52 may be seen by the user with an eye disease such asmacular degeneration. The second grid 52 seen by the user may include alost area 53 where the grid is not visible, and a distorted area 54where the grid appears distorted.

For example, a preferential hyperacuity perimeter (PHP) test may havebeen used to diagnose visual impairment of a user. The PHP test is atest related to the ability of the user to identify misaligned lines ordots.

A user with visual impairment due to an eye disease such as maculardegeneration may not recognize misaligned small dots. The PHP test isperformed by presenting a plurality of aligned small dots and checkingthe ability of the user to recognize misaligned dots among the pluralityof small dots.

In general, visual impairment due to an eye disease such as maculardegeneration requires continuous monitoring of eye conditions, andself-diagnosis using specific equipment may not be easily performed bythe user whereas self-diagnosis using test paper may be less accurate.

According to an embodiment of the disclosure, a user may easily andaccurately self-diagnose visual impairment while using an AR device. Inaddition, when visual impairment is diagnosed, the AR device may provideassistance to the user to minimize inconvenience of visual impairment,while providing an AR service.

FIG. 1B is a view for describing an embodiment of the disclosure.

An electronic device 100 according to an embodiment of the disclosuremay be an AR device. For example, as illustrated in FIG. 1B, theelectronic device 100 may be a device implemented in the form of glasseswhich are wearable on the face of a user. The electronic device 100 maybe a device implemented in the form of goggles, a helmet, or a hat thatmay be worn on the head of the user, but is not limited thereto.

As illustrated in FIG. 1B, while the user is wearing the electronicdevice 100 provided in the form of glasses which are wearable on theface of the user, when the electronic device 100 provides a diagnosisimage including a certain pattern or dots through a display 140 (seeFIG. 2) in order to diagnose visual impairment of the user, thediagnosis image may be seen like a first image 102 by a user with normalvision and be seen like a second image 104 by a user with visualimpairment due to aging of the eyes or an eye disease such as maculardegeneration. For example, the second image 104 seen by the user withvisual impairment may include a lost area 105 which is not visible tothe eyes of the user, and a distorted area 106 where a partial patternappears distorted.

According to an embodiment of the disclosure, an entire visual field ofthe eyes of the user may include an entire display area on a waveguide142 (see FIG. 2) of the electronic device 100. According to anembodiment of the disclosure, a point of the entire visual field maycorrespond to a point on the waveguide 142 (see FIG. 2) when a specificuser wearing the electronic device 100 sees a real scene or a virtualimage through the waveguide 142.

An impaired area according to an embodiment of the disclosure may referto at least a partial area of the entire visual field which is notvisible to the eyes of the user or appears distorted. The impaired areamay include at least one of a lost area or a distorted area.

The lost area according to an embodiment of the disclosure may refer toat least a partial area which is not visible to the eyes of the user.

The distorted area according to an embodiment of the disclosure mayrefer to at least a partial area which appears distorted.

A normal area according to an embodiment of the disclosure may refer toan area excluding the impaired area from the entire visual field.

FIG. 10 is a view for describing a visual impairment diagnosis methodaccording to an embodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may detect, through visual impairment diagnosis, an impaired area in anentire visual field of the eyes of a user wearing the electronic device100.

For example, as illustrated in FIG. 10, the electronic device 100 maysequentially output diagnosis images 113, 115, and 117 in which a targetimage (e.g., a star-shaped dot) for attracting the eyes of the user isdisplayed at different locations 114, 116, and 118, and induce the userto look at the target image. When the target image is output at thedifferent locations 114, 116, and 118, the electronic device 100 maystore gaze information detected using a gaze tracking sensor 152 (seeFIG. 2), in a memory 130 (see FIG. 2) in the form of a table.

The gaze information according to an embodiment of the disclosure may beinformation obtained by the gaze tracking sensor 152 (see FIG. 2) of theelectronic device 100, and include at least one of a gaze direction ofthe eyes of the user, position of the pupils of the eyes of the user, orcoordinates of the centers of the pupils.

The electronic device 100 may determine whether a gaze of the eyes ofthe user is directed to a location where the target image is output,based on the gaze information obtained by the gaze tracking sensor 152(see FIG. 2). The electronic device 100 may determine the impaired areabased on the location of the output target image when the gaze of theeyes of the user is not directed to the location where the target imageis output.

For example, when the electronic device 100 outputs the target image ata specific location 118 of the entire visual field, and when the gaze ofthe eyes of the user is not directed to a specific location 118 wherethe target image is output, and is directed to a default location 112,an area including the specific location 118 may be determined as theimpaired area.

According to an embodiment of the disclosure, the electronic device 100may generate a vision map by determining the impaired area of vision ofthe user by using the obtained gaze information, and store the visionmap in the memory 130.

The vision map according to an embodiment of the disclosure may refer todata including information about locations and ranges of a normal areaand an impaired area in an entire visual field of the eyes of a specificuser.

For example, referring to FIG. 10, the vision map 120 may includeinformation about locations and ranges of a normal area 122 and animpaired area 123 in the entire visual field of the user.

According to an embodiment of the disclosure, the electronic device 100may provide a user-recognizable interface related to the impaired areanot clearly visible to the user wearing the electronic device 100, byusing the vision map generated through visual impairment diagnosis. Forexample, the electronic device 100 may display a virtual imageindicating a location and range of the impaired area, in the normal areaof vision of the user. The electronic device 100 may move an objectincluded in the impaired area not visible to the user, to be displayedin the normal area of vision of the user, by using the vision map.

According to an embodiment of the disclosure, a user with visualimpairment due to an eye disease such as macular degeneration may easilydiagnose visual impairment and continuously monitor impairment by usingthe electronic device 100 without using additional equipment.

In addition, according to an embodiment of the disclosure, theelectronic device 100 may provide a user-specific AR service on thebasis of previously obtained visual impairment information of the user.The electronic device 100 may provide assistance to the user torecognize the impaired area and to see an object, which is located inthe impaired area and thus is not accurately visible to the user, in thenormal area, thereby minimizing inconvenience caused by visualimpairment.

A method, performed by the electronic device 100, of diagnosing visualimpairment of the eyes of the user, a method, performed by theelectronic device 100, of providing assistance to the user with visualimpairment, and examples thereof will be described below with referenceto the drawings.

FIG. 2 is a block diagram of the electronic device 100 according to anembodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may be an AR device including a communication function and a dataprocessing function to provide AR images, but is not limited thereto.

Referring to FIG. 2, the electronic device 100 according to anembodiment of the disclosure may include the memory 130, a processor120, the display 140, a varifocal lens 145, a sensor 150, a cameramodule 175, a communicator 180, an audio outputter 185, a vibrationmotor 187, a microphone 188, an a user inputter 189. However, not allelements illustrated in FIG. 2 are essential elements of the electronicdevice 100. The electronic device 100 may be implemented with more orfewer elements than the elements illustrated in FIG. 2.

The processor 120 of the electronic device 100 may execute programsstored in the memory 130 to control the display 140, the varifocal lens145, the sensor 150, the camera module 175, the communicator 180, theaudio outputter 185, the vibration motor 187, the microphone 188, andthe user inputter 189.

The memory 130 according to an embodiment of the disclosure may storethe programs to be executed by the processor 120, and store data inputto or to be output from the electronic device 100.

The memory 130 may include at least one type of storage medium amongflash memory, a hard disk, a multimedia card micro, a memory card (e.g.,a secure digital (SD) or extreme digital (XD) memory card), randomaccess memory (RAM), static random access memory (SRAM), read-onlymemory (ROM), electrically erasable programmable read-only memory(EEPROM), programmable read-only memory (PROM), magnetic memory, amagnetic disc, and an optical disc.

The programs stored in the memory 130 may be classified into a pluralityof software modules, for example, a diagnosis module 131 and anassistance module 132, depending on functions thereof, but are notlimited thereto and may include only some of the above-mentionedsoftware modules or further include other software modules.

The processor 120 controls overall operations of the electronic device100. The processor 120 may execute instructions or programs stored inthe memory 130 to control operations or functions performed by theelectronic device 100.

According to an embodiment of the disclosure, the processor 120 mayinclude one or more processors. The processor 120 may include at leastone type of hardware among, for example, a central processing unit(CPU), a microprocessor, a graphics processing unit (GPU), anapplication-specific integrated circuit (ASIC), a digital signalprocessor (DSP), a digital signal processing device (DSPD), aprogrammable logic device (PLD), and a field programmable gate array(FPGA), but is not limited thereto.

The processor 120 may execute the diagnosis module 131 stored in thememory 130 to detect an impaired area of vision of a user.

In order to detect the impaired area of vision of the user, theprocessor 120 may control an optical engine 141 to output a diagnosisimage including a target image through the waveguide 142.

According to an embodiment of the disclosure, the diagnosis image mayinclude the target image configured as a certain-shaped dot or a certainpattern. For example, the certain-shaped dot includes a star-shaped,X-shaped, or plus-shaped dot, but is not limited thereto. The certainpattern includes a plurality of dots aligned at certain intervals, anarbitrary closed curve, or a wave pattern, but is not limited thereto.

According to an embodiment of the disclosure, the impaired area mayinclude at least one of a lost area or a distorted area.

According to an embodiment of the disclosure, in order to detect thelost area, the processor 120 may use a diagnosis image including a firsttarget image configured as a certain-shaped dot.

The processor 120 may control the optical engine 141 to output the firsttarget image through the waveguide 142 sequentially at preset differentlocations of an entire visual field of the eyes of the user at presettime intervals.

According to an embodiment of the disclosure, the preset differentlocations may include at least one of a plurality of locations spacedapart from each other at certain intervals on the waveguide 142.

The processor 120 may output, through the audio outputter 185, guidancesound for instructing the user wearing the electronic device 100 to lookat the first target image displayed through the waveguide 142.

The processor 120 may obtain gaze information of the eyes of the usercorresponding to a location of the output first target image by usingthe gaze tracking sensor 152 when the first target image is outputsequentially at the preset different locations. The processor 120 maydetermine whether a gaze direction of the eyes of the user converges onthe location where the first target image is output, based on theobtained gaze information. According to a result of the determination,the processor 120 may determine an impaired area of the entire visualfield based on the location of the output first target image.

When vision of the eyes of the user includes a lost area, and when thetarget image is displayed at a location corresponding to the lost areain the entire visual field, the eyes of the user may not recognize thetarget image and thus a gaze of the eyes of the user will not bedirected to the location where the target image is displayed.

While the first target image (e.g., a star-shaped dot) is being outputsequentially at the preset different locations, the processor 120 maydetermine a certain area including the location where the first targetimage is output when the gaze direction of the eyes of the user does notconverge on the location where the first target image is output, as thelost area.

The processor 120 may generate a vision map related to the impairedarea, and a normal area excluding the impaired area from the entirevisual field, based on the impaired area including the determined lostarea, and store the vision map in the memory 130.

In order to detect the lost area of each of the left and right eyes ofthe user, the processor 120 may provide the same diagnosis imageincluding the target image sequentially to a left-eye waveguide 142L(see FIG. 3A) and a right-eye waveguide 142R (see FIG. 3A). Theprocessor 120 may determine that the user may not recognize thediagnosis image provided on the left-eye waveguide 142L (see FIG. 3A) orthe right-eye waveguide 142R (see FIG. 3A), and determine visualimpairment of the left or right eye of the user.

According to an embodiment of the disclosure, in order to detect thedistorted area, the processor 120 may use a diagnosis image including asecond target image configured as a certain pattern.

The processor 120 may control the optical engine 141 to output thesecond target image through the waveguide 142. For example, theprocessor 120 may output a plurality of dots aligned in a certainpattern sequentially at preset different locations of the entire visualfield. The preset different locations may include at least one of aplurality of locations spaced apart from each other at certain intervalson the waveguide 142.

For example, the processor 120 may output an arbitrary closed curveincluding a wave pattern by gradually reducing the size thereof from theperiphery to the center of vision of the user.

According to an embodiment of the disclosure, the processor 120 mayoutput, through the audio outputter 185, guidance information related tothe second target image. For example, the processor 120 may output soundfor instructing to trace the pattern of the second target imageconfigured as a dashed line with a finger.

According to an embodiment of the disclosure, the processor 120 mayobtain a gesture input of the user by using a depth sensor 153. Forexample, the processor 120 may obtain a gesture input of the userwearing the electronic device 100 for tracing a pattern of alignment ofthe plurality of dots output on the waveguide 142 with a finger whilelooking at the plurality of dots.

The processor 120 may obtain a user input pattern based on the obtainedgesture input of the user.

The processor 120 may compare the second target image configured as thecertain pattern to the obtained user input pattern. The processor 120may determine the distorted area based on a location of the outputsecond target image according to a result of the comparison.

When vision of the eyes of the user includes a distorted area, and whenthe certain pattern, e.g., the target image in which the plurality ofdots are aligned, is displayed at a location corresponding to thedistorted area of the entire visual field, the pattern in which theplurality of dots are aligned may appear distorted and thus the userinput pattern may not match the pattern of the target image.

The processor 120 may determine an area where the second target imagedoes not match the obtained user input pattern, and determine thedistorted area based on the area where the second target image does notmatch the user input pattern.

The processor 120 may generate a vision map related to the impairedarea, and a normal area excluding the impaired area from the entirevisual field, based on the impaired area including the determineddistorted area, and store the vision map in the memory 130.

The processor 120 may compare the second target image configured as thecertain pattern to the obtained user input pattern, and calculate adegree of distortion in the distorted area, based on a distance betweenthe second target image and the obtained user input pattern.

In order to detect the distorted area of each of the left and right eyesof the user, the processor 120 may provide the diagnosis image includingthe target image on any one of the left-eye waveguide 142L (see FIG. 3A)and the right-eye waveguide 142R (see FIG. 3A). The processor 120 maydetermine that the user may not accurately recognize the diagnosis imageprovided on the left-eye waveguide 142L (see FIG. 3A) or the right-eyewaveguide 142R (see FIG. 3A), and determine visual impairment of theleft or right eye of the user.

The processor 120 may execute the assistance module 132 stored in thememory 130 to provide assistance to the user to recognize the impairedarea.

According to an embodiment of the disclosure, the processor 120 maycontrol the optical engine 141 to output a guidance image for notifyingthe user of the impaired area through the waveguide 142, based on thepreviously generated vision map.

For example, the processor 120 may display a virtual guidance imageindicating the impaired area of vision of the user, in the normal areaadjacent to the impaired area of vision of the user.

As such, when the user sees a real scene through the waveguide 142, theuser may recognize that a real world object of the real scene, which islocated in a direction and at a distance where a guidance image isdisplayed, is not accurately visible due to visual impairment of theuser. In addition, according to an embodiment of the disclosure, theprocessor 120 may control an object included in the impaired area to bedisplayed in the normal area of vision of the user by using a prism modeof the varifocal lens 145 such that the user may recognize the objectincluded in the impaired area.

According to an embodiment of the disclosure, when the user wearing theelectronic device 100 sees a real scene through the waveguide 142, theprocessor 120 may detect an object included in the impaired area on anentire display area of the waveguide 142, based on the previouslygenerated vision map. The processor 120 may capture an image of a realscene in front of the electronic device 100 by using the camera module175, and perform object recognition on the captured image by using acertain object recognition algorithm. The processor 120 may detect anobject included in an area of the captured image corresponding to theimpaired area of vision of the user. The processor 120 may detect adepth of at least one real world object included in the real scene infront of the electronic device 100 by using the depth sensor 153. Theprocessor 120 may detect the object included in the area correspondingto the impaired area of vision of the user, based on the detected depthof the at least one real world object.

The processor 120 may apply the prism mode to a specific area of thevarifocal lens 145 corresponding to the impaired area, in such a mannerthat the detected object is displayed in the normal area of vision ofthe user.

According to an embodiment of the disclosure, the prism mode is a modein which refractive power of the specific area of the varifocal lens 145is changed. The refractive power of the varifocal lens 145 may bechanged by changing orientation of liquid crystal (LC) molecules in thevarifocal lens 145.

When the prism mode is applied to the varifocal lens 145, orientation ofthe LC molecules may be changed by applying a voltage to the specificarea of the varifocal lens 145 corresponding to the impaired area ofvision of the user to have a phase profile corresponding to the prismmode, and thus the refractive power may be changed. As such, a path oflight passing through the specific area of the varifocal lens 145 may bechanged.

The processor 120 may exert control to move an image of the objectincluded in the impaired area to a location corresponding to the normalarea of vision of the user, by applying the prism mode to the specificarea of the varifocal lens 145 corresponding to the impaired area.

As such, when the user wearing the electronic device 100 sees the realscene, the user may recognize the object, which is included in theimpaired area of vision of the user and thus is not visible to the user,at another location.

The display 140 may output information processed by the processor 120.For example, the display 140 may display a virtual object.

According to an embodiment of the disclosure, the display 140 mayprovide an AR image. The display 140 according to an embodiment of thedisclosure may include the waveguide 142 and the optical engine 141.

The waveguide 142 may be made of a transparent material, and a partialarea of a rear surface thereof may available for view by the user whenthe electronic device 100 is worn. The waveguide 142 may be configuredas a transparent monolayer or multilayer flat panel capable ofreflecting and propagating light therein. The waveguide 142 may face anexit surface of the optical engine 141 to receive light of a virtualimage projected from the optical engine 141. Herein, the transparentmaterial refers to a material through which light may pass, and may ormay not have a transparency of 100% and may have a certain color.

In an embodiment of the disclosure, because the waveguide 142 is made ofa transparent material, the user may see not only a virtual object of avirtual image but also an external real scene through the display 140,and thus the waveguide 142 may also be called a see-through display. Thedisplay 140 may provide an AR image by outputting a virtual object of avirtual image through the waveguide 142.

The varifocal lens 145 may be mounted in the electronic device 100 toprovide assistance to the user with visual impairment. The varifocallens 145 may be aligned to overlap with the waveguide 142 to face theeyes of the user. The varifocal lens 145 may be generally implemented asa liquid lens or an LC lens. For example, the varifocal lens 145 may beimplemented as a liquid lens in which a transparent fluid is surroundedby a flexible plastic membrane. The fluid in the varifocal lens 145 maymove according to an electrical signal applied to the varifocal lens145, and thus refractive power of the varifocal lens 145 may be changed.As another example, the varifocal lens 145 may be implemented as an LClens in which a transparent electrode is provided on both surfaces of atransparent LC layer. Orientation of liquid crystals in the LC layer maybe changed according to an electrical signal applied to the transparentelectrode, and thus a path of light passing through the LC lens may bechanged and the refractive power of the varifocal lens 145 may bechanged.

For example, an electrical signal or voltage value to be applied to anelectrode may be preset in such a manner that the refractive power ofthe varifocal lens 145 corresponds to a diopter value (e.g., . . . −3D,−2D, −1D, 0, 1D, 2D, 3D . . . ), and refractive power of a correspondingdiopter value may be applied to the varifocal lens 145 when anelectrical signal or voltage is applied to the electrode. However, therefractive power of the varifocal lens 145 is not limited thereto and,for example, the electrical signal or voltage value to be applied to theelectrode may be preset in such a manner that the refractive power ofthe varifocal lens 145 may be changed to successive values.

According to an embodiment of the disclosure, a prism mode may beapplied to a specific area of the varifocal lens 145. The prism mode isa mode in which refractive power of the specific area of the varifocallens 145 is changed. The refractive power of the varifocal lens 145 maybe changed by changing orientation of LC molecules in the varifocal lens145.

According to an embodiment of the disclosure, when the prism mode isapplied to the varifocal lens 145, orientation of the LC molecules maybe changed by applying a voltage to the specific area of the varifocallens 145 to have a phase profile corresponding to the prism mode, andthus the refractive power may be changed. As such, a path of lightpassing through the specific area of the varifocal lens 145 may bechanged.

According to an embodiment of the disclosure, the prism mode may beapplied to the specific area of the varifocal lens 145 corresponding tothe impaired area of vision of the user. By changing the path of lightpassing through the specific area of the varifocal lens 145 to which theprism mode is applied, an image of a real world object included in theimpaired area may be displayed in the normal area of vision of the user.In general, a prism may cause dispersion of light by diffraction (seethe illustration on the right hand side of FIG. 5). In some embodiments,the varifocal lens 145 in prism mode disperses light of the real worldobject away from the impaired area and into the normal area of vision.The user thus obtains information of the real world object.

When the electronic device 100 is a glass-type device, the varifocallens 145 may include a left-eye varifocal lens and a right-eye varifocallens.

The sensor 150 may include a motion sensor 151, the gaze tracking sensor152, and the depth sensor 153.

The motion sensor 151 may be an inertial measurement unit (IMU). The IMUmay be a combination of sensors configured to detect motion, i.e.,changes in position and orientation, of an object in the 3-dimensionalspace. For example, the combination of sensors may include anaccelerometer, an angular speedometer, a magnetometer, and a gyroscope.

The motion sensor 151 may include at least one of an accelerationsensor, a magnetic sensor, or a gyroscope sensor.

According to an embodiment of the disclosure, when the user wearing theelectronic device 100 moves the head according to the pattern of thetarget image included in the diagnosis image for diagnosing visualimpairment of the user, the motion sensor 151 may sense a head motionpattern of the user.

The gaze tracking sensor 152 may detect gaze information of the eyes ofthe user. According to an embodiment of the disclosure, the gazeinformation may include at least one of a gaze direction of the eyes ofthe user, position of the pupils of the eyes of the user, or coordinatesof the centers of the pupils.

The gaze tracking sensor 152 may provide light to the eyes (e.g., theleft or right eye) of the user and sense the amount of light reflectedfrom the eyes of the user. The gaze tracking sensor 152 may detect thegaze direction of the eyes of the user, the position of the pupils ofthe eyes of the user, or the coordinates of the centers of the pupils,based on the sensed amount of light.

Alternatively, the gaze tracking sensor 152 may provide light to theeyes of the user and capture an image of the eyes of the user. The gazetracking sensor 152 may detect the gaze direction of the eyes of theuser, the position of the pupils of the eyes of the user, or thecoordinates of the centers of the pupils, based on the captured image ofthe eyes of the user.

The depth sensor 153 may obtain depth information of one or more objectsincluded in the real world. The depth information may correspond to adistance from the depth sensor 153 to a specific object. The depth valuemay increase in proportion to the distance from the depth sensor 153according to an embodiment of the disclosure to the specific object.

The depth sensor 153 according to an embodiment of the disclosure mayobtain the depth information of the objects in various manners. Forexample, the depth sensor 153 may obtain the depth information by usingat least one of a time of flight (TOF) method, a structured lightmethod, or a stereo image method. The depth sensor 153 using the stereoimage method generally includes two or more cameras.

According to an embodiment of the disclosure, the depth sensor 153 maysense depth information of a real world object included in a real sceneseen through the waveguide 142 by the user wearing the electronic device100. The processor 120 may obtain information indicating whether thereal world object is present in front of the electronic device 100, adirection and distance of the real world object, etc., based on thedepth information of the real world object sensed by the depth sensor153.

The depth sensor 153 may also sense depth information of fingers of theuser wearing the electronic device 100. The processor 120 may obtain agesture input of a hand of the user by recognizing a shape or a motionpattern of the hand of the user, based on the depth information of thefingers sensed by the depth sensor 153.

The camera module 175 may capture an image of an ambient environment ofthe electronic device 100. The camera module 175 may obtain still imagesor video frames by using an image sensor when an application requiringan image capture function is executed.

The images captured using the image sensor may be processed by theprocessor 120 or a separate image processor (not shown). The capturedimages may be displayed through the display 140.

The images processed by the processor 120 or the separate imageprocessor (not shown) may be stored in the memory 130 or transmittedthrough the communicator 180 to an external device. Two or more cameramodules 175 may be provided depending on the configuration of theelectronic device 100.

The communicator 180 may include one or more elements for enablingcommunication between the electronic device 100 and an external server(not shown) or an external device (not shown).

For example, the communicator 180 may include a short-range wirelesscommunicator and a mobile communicator.

The short-range wireless communicator may include a Bluetoothcommunicator, a near-field communication (NFC)/radio-frequencyidentification (RFID) communicator, a wireless local area network (WLAN)(or Wi-Fi) communicator, a Zigbee communicator, an infrared dataassociation (IrDA) communicator, a Wi-Fi direct (WFD) communicator, aultra-wideband (UWB) communicator, or an Ant+ communicator, but is notlimited thereto.

The mobile communicator transmits and receives wireless signals to andfrom at least one of a base station, an external device, or a server ina mobile communication network. Herein, the wireless signals may includevarious types of data related to transmission and reception of voicecall signals, video call signals, or text/multimedia messages.

The audio outputter 185 outputs audio data received from thecommunicator 180 or stored in the memory 130. The audio outputter 185outputs an audio signal related to a function performed by theelectronic device 100 (e.g., call signal reception sound, messagereception sound, or notification sound).

The audio outputter 185 according to an embodiment of the disclosure mayinclude a speaker or a buzzer. The audio outputter 185 according to anembodiment of the disclosure may be implemented in the form of earphonesmounted on or detachable from the electronic device 100. The audiooutputter 185 according to an embodiment of the disclosure may outputaudio data in a bone conduction manner.

The audio outputter 185 according to an embodiment of the disclosure mayoutput guidance information for instructing the user wearing theelectronic device 100 to look at the target image displayed through thewaveguide 142. The audio outputter 185 may output guidance informationfor instructing the user wearing the electronic device 100 to trace thepattern of the target image displayed through the waveguide 142 with afinger, but is not limited thereto.

The vibration motor 187 may output a vibration signal. For example, thevibration motor 187 may output a vibration signal corresponding tooutput of audio or video data (e.g., call signal reception sound ormessage reception sound). The vibration motor 187 may output a vibrationsignal when a user input is received from the user inputter 189. Thevibration motor 187 may provide a notification with vibration when theelectronic device 100 operates in a vibration mode.

The microphone 188 receives an external audio signal and processes theaudio signal into electrical voice data. For example, the microphone 188may receive the audio signal from an external device or user. Themicrophone 188 may receive a voice input of the user for controlling theelectronic device 100. The microphone 188 may use various noisecancellation algorithms to cancel noise added while receiving anexternal audio signal.

The user inputter 189 refers to a means by which a user inputs data forcontrolling the electronic device 100. For example, the user inputter189 may include at least one of a keypad, a dome switch, a touchpad(e.g., a capacitive overlay, resistive overlay, infrared beam, surfaceacoustic wave, integral strain gauge, or piezoelectric touchpad), a jogwheel, or a jog switch, but is not limited thereto.

FIG. 3A is a view showing an example of the electronic device 100according to an embodiment of the disclosure.

FIG. 3A is a view showing an example of an AR device including thevarifocal lens 145, according to an embodiment of the disclosure. Theelectronic device 100 of FIG. 2 may be implemented as, for example, aglass-type display device including a glass-type body wearable by a useras illustrated in FIG. 3A and indicated with item number 100, but is notlimited thereto.

The glass-type body may include a frame 110 and glass legs 190. Theglass legs 190 may include a left leg 190L and a right leg 190R, and beconnected to both end pieces of the frame 110.

The varifocal lens 145 and the waveguide 142 may be provided on theframe 110. The varifocal lens 145 may include a left-eye varifocal lens145L and a right-eye varifocal lens 145R. The waveguide 142 may beconfigured to receive projected light input to an input area and outputat least a part of the input light from an output area. The waveguide142 may include the left-eye waveguide 142L and the right-eye waveguide142R.

The left-eye varifocal lens 145L and the left-eye waveguide 142L may beprovided at a location corresponding to the left eye of the user, andthe right-eye varifocal lens 145R and the right-eye waveguide 142R maybe provided at a location corresponding to the right eye of the user.For example, the left-eye varifocal lens 145L and the left-eye waveguide142L, or the right-eye varifocal lens 145R and the right-eye waveguide142R may be attached to each other, but are not limited thereto.

The optical engine 141 including a projector that projects lightcontaining an image may include a left-eye optical engine 141L and aright-eye optical engine 141R. The left-eye optical engine 141L and theright-eye optical engine 141R may be located at both end pieces of theframe 110. Light emitted from the optical engine 141 may be displayedthrough the waveguide 142.

The electronic device 100 may include the gaze tracking sensor 152 totrack a gaze of the user. The gaze tracking sensor 152 according to anembodiment of the disclosure may include a first gaze tracking sensor152L for tracking a gaze of the left eye of the user, and a second gazetracking sensor 152R for tracking a gaze of the right eye of the user.

Two example camera modules 175 are indicated in FIG. 3A. Also, processor120, memory 130, sensor 150, communicator 180, audio outputter 185,vibration motor 187 and microphone 188 are indicated in FIG. 3A (theseitems do not need to be co-located in electronic device 100). The cameramodules 175 may be located for capture of good-quality images for stereoprocessing. The processor 120, memory 130, sensor 150, communicator 180,audio outputter 185, vibration motor 187 and microphone 188 may beconfigured and located in, on or with electronic device 100 based onmanufacturing, ergonomic and cost factors.

FIG. 3B is a view for describing the optical engine 141 and thewaveguide 142, according to an embodiment of the disclosure.

The optical engine 141 may be configured to generate light of a virtualimage, and include, for example, an image panel and a projectorincluding a projection optical system.

The optical engine 141 may include a light source for outputting light,an image panel for forming a 2-dimensional virtual image by using thelight output from the light source, and a projection optical system forprojecting light of the virtual image formed on the image panel. Thelight source may be an optical part for emitting light, and generatelight by adjusting RGB colors. The light source may be configured as,for example, a light-emitting diode (LED). The image panel may beconfigured as a reflective image panel for modulating and reflecting thelight emitted from the light source into light containing a2-dimensional image. The reflective image panel may be, for example, adigital micromirror device (DMD) panel, a liquid crystal on silicon(LCoS) panel, or another known reflective image panel. The projectionoptical system may be an element for projecting the light containing theimage and reflected by the image panel, onto the waveguide 142, andinclude one or more projection lenses.

The optical engine 141 may obtain image data configuring a virtual imagefrom the processor 120, generate a virtual image based on the obtainedimage data, and project the light configuring the virtual image andoutput from the light source, through an exit surface 1140 onto thewaveguide 142. The processor 120 may provide image data including RGBcolor values and brightness values of a plurality of pixels configuringa virtual image to the optical engine 141, and the optical engine 141may project light configuring the virtual image onto the waveguide 142by controlling the light source according to the RGB color values andthe brightness values of the plurality of pixels. The optical engine 141may project the virtual image by using a transmissive projectiontechnology where a light source is modulated by an optically activematerial and backlit with white light.

The waveguide 142 may be made of a transparent material, and a partialarea of a rear surface thereof may be available for view by a user whenthe electronic device 100 is worn. The rear surface of the waveguide 142refers to a surface facing the eyes of the user when the electronicdevice 100 is worn, and a front surface of the waveguide 142 refers to asurface opposite to the rear surface (i.e., a surface away from the eyesof the user).

According to an embodiment of the disclosure, the waveguide 142 may beconfigured as a transparent monolayer or multilayer flat panel capableof reflecting and propagating light therein. The waveguide 142 mayinclude a first area 1110 facing the exit surface 1140 of the opticalengine 141 to receive light configuring a projected virtual image VI, asecond area 1120 for propagating the light configuring the virtual imageVI and incident on the first area 1110, and a third area 1130 foroutputting the light of the virtual image VI propagated in the secondarea 1120, toward the eyes of the user.

Each of the first area 1110, the second area 1120, and the third area1130 may include a diffraction grating for changing a path of the lightconfiguring the virtual image VI. The waveguide 142 may guide light bychanging a propagation path of the light of the virtual image VI byusing the diffraction gratings on the first area 1110, the second area1120, and the third area 1130, and ultimately outputting reflected lightof the virtual image VI from the third area 1130 to the eyes of theuser.

FIG. 4A is a view for describing a method of operating the gaze trackingsensor 152, according to an embodiment of the disclosure.

FIG. 4A is a view for describing a method of tracking a gaze of a userbased on the amount of light reflected from the eyes of the user.

Because the first gaze tracking sensor 152L and the second gaze trackingsensor 152R according to an embodiment of the disclosure have the samestructure and operate in the same manner, the first gaze tracking sensor152L will be representatively described in relation to FIG. 4A.

Referring to FIG. 4A, the first gaze tracking sensor 152L according toan embodiment of the disclosure may include a light emitter 301 forproviding light to an eye of the user, and a sensor 302 for sensinglight. The light emitter 301 may include a light source for providinglight, and a scanning mirror for controlling the direction of the lightprovided from the light source. The scanning mirror may control thelight provided from the light source to proceed toward an eye 320 (e.g.,a cornea 310) of the user wearing the electronic device 100. Thescanning mirror may include a structure capable of mechanically changinga reflection angle to reflect the light provided from the light sourcetoward the eye 320 of the user, and scan an area including the cornea310 by using the light provided from the light source according to thechanged reflection angle.

The sensor 302 may sense light reflected from the eye 320 of the user,and measure the amount of the sensed light. For example, when light isreflected from the center of the cornea 310 of the user, the amount ofthe light sensed by the sensor 302 may be the highest. As such, thefirst gaze tracking sensor 152L may determine a gaze direction 340 ofthe eye of the user based on a point where the light is incident on andreflected from the eye of the user when the amount of the light sensedby the sensor 302 is the highest. For example, the first gaze trackingsensor 152L may determine the direction 340 of a virtual line connectingthe center of the eye 320 of the user to a point 330 where the light isincident on and reflected from the eye of the user when the amount ofthe light is the highest, as a gaze direction of the eye of the user(e.g., the left eye of the user). However, the gaze directiondetermination method is not limited thereto.

The second gaze tracking sensor 152R may also determine a gaze directionof an eye (e.g., the right eye) of the user in the same manner as thatdescribed above in relation to FIG. 4A.

FIG. 4B is a view for describing a method of operating the gaze trackingsensor 152, according to another embodiment of the disclosure. The firstgaze tracking sensor 152L according to another embodiment of thedisclosure may include a light emitter 351 and an image capturer 352.

FIG. 4B is a view for describing a method of tracking a gaze of a userbased on a position of light reflected from an eye of the user.

The light emitter 351 according to an embodiment of the disclosure mayinclude, for example, an infrared light-emitting diode (IR LED). Asillustrated in FIG. 4B, the light emitter 351 may include a plurality ofLEDs provided at different locations. The light emitter 351 may providelight (e.g., infrared light) to an eye of the user to capture an imageof the eye of the user. When the light is provided to the eye of theuser, the light may be reflected on the eye of the user.

The image capturer 352 may include at least one camera and, in thiscase, the at least one camera may include an infrared (IR) camera. Theelectronic device 100 may track a gaze of the eye of the user (e.g., theleft eye of the user) by using an eye image of the user captured by theimage capturer 352. For example, the first gaze tracking sensor 152L maytrack the gaze of the user by detecting a pupil and reflected light inthe eye image of the user. The first gaze tracking sensor 152L maydetect positions of the pupil and the reflected light in the eye imageof the user, and determine a gaze direction of the eye of the user basedon the relationship between the position of the pupil and the positionof the reflected light.

For example, the first gaze tracking sensor 152L may detect a pupil 370and reflected light 381 in a captured first eye image 361, and determinea gaze direction 391 of the eye of the user based on the relationshipbetween a position of the pupil 370 and a position of the reflectedlight 381. In the same manner, the first gaze tracking sensor 152L maydetect the pupil 370 and reflected light 382, 383, 384, or 385 in eachof second to fifth eye images 362, 363, 364, and 365, and determine agaze direction 392, 393, 394, or 395 of the eye of the user based on therelationship between a position of the pupil 370 and a position of thereflected light 382, 383, 384, or 385.

The second gaze tracking sensor 152R may also determine a gaze directionof an eye (e.g., the right eye) of the user in the same manner as thatdescribed above in relation to FIG. 4B.

FIG. 5 is a view showing an example in which refractive power of thevarifocal lens 145 is changed, according to an embodiment of thedisclosure.

Referring to FIG. 5, the varifocal lens 145 of FIG. 3A may beimplemented, for example, to include an LC layer 610 including LCmolecules 612 having a variable orientation angle. In the LC layer 610of the varifocal lens 145, a control voltage modulated to have aspecific phase profile may be applied to an electrode 30 and thus theorientation angle of the LC molecules 612 provided at a specificlocation in an active area may be changed. When the orientation angle ofthe LC molecules 612 provided in a specific area of the LC layer 610 ischanged, a refractive index of light passing through the LC molecules612 may be changed. When the refractive index of the light is changed,the refractive power of the varifocal lens 145 may be changed and thus apath of the light passing through the varifocal lens 145 may be changed,thereby changing a vergence. The vergence is an index indicating adegree by which the light passing through the varifocal lens 145converges or diverges. The vergence may be adjusted according to therefractive power of the varifocal lens 145.

FIG. 6 is a view for describing a prism mode of the varifocal lens 145,according to an embodiment of the disclosure.

The prism mode according to an embodiment of the disclosure is a mode inwhich refractive power of a specific area of the varifocal lens 145 ischanged. The refractive power of the varifocal lens 145 may be changedby changing orientation of LC molecules in the varifocal lens 145.

FIG. 6 shows an example in which a lens mode is applied to a first area621 of the varifocal lens 145 and a prism mode is applied to a secondarea 622 of the varifocal lens 145.

According to an embodiment of the disclosure, when the lens mode isapplied to the first area 621 of the varifocal lens 145, a voltage maybe applied to the first area 621 to have a phase profile 631corresponding to the lens mode.

According to an embodiment of the disclosure, when the prism mode isapplied to the second area 622 of the varifocal lens 145, orientation ofthe LC molecules may be changed by applying a voltage to the second area622 to have a phase profile 632 corresponding to the prism mode, andthus the refractive power may be changed. As such, a path of lightpassing through the second area 622 may be changed.

For example, the prism mode may be applied to the second area 622 of thevarifocal lens 145 corresponding to an impaired area of vision of auser. By changing the path of light passing through the second area 622of the varifocal lens 145 to which the prism mode is applied, a locationfor displaying an image of a real world object included in the impairedarea may be changed from a first location 651 to a second location 652.

According to an embodiment of the disclosure, when a user with visualimpairment sees a real scene through the waveguide 142, the user mayrecognize the real world object, which is included in the impaired areaof vision of the user, at another location corresponding to a normalarea of vision of the user.

FIG. 7 is a flowchart of a method, performed by the electronic device100, of diagnosing visual impairment based on obtaining of gazeinformation, according to an embodiment of the disclosure.

In operation S701 of FIG. 7, the electronic device 100 may control theoptical engine 141 to output a first target image through the waveguide142 at preset different locations.

According to an embodiment of the disclosure, a diagnosis image fordiagnosing visual impairment of a user may include a target imageconfigured as a certain-shaped dot or a certain pattern. For example,the first target image may be a star-shaped, X-shaped, or plus-shapeddot, but is not limited thereto.

According to an embodiment of the disclosure, the preset differentlocations may include at least one of a plurality of locations spacedapart from each other at certain intervals on the waveguide 142.

The electronic device 100 may control the optical engine 141 to outputthe first target image through the waveguide 142 sequentially at presetdifferent locations of an entire visual field of the eyes of the user atpreset time intervals.

According to an embodiment of the disclosure, the electronic device 100may output, through the audio outputter 185, guidance information forinstructing the user wearing the electronic device 100 to look at thefirst target image displayed through the waveguide 142 (e.g., ‘Look atthe displayed dot’).

In operation S702 of FIG. 7, the electronic device 100 may obtain gazeinformation of the eyes of the user corresponding to a location of theoutput first target image by using the gaze tracking sensor 152 when thefirst target image is output.

According to an embodiment of the disclosure, the electronic device 100may obtain the gaze information corresponding to the location of theoutput first target image by using the gaze tracking sensor 152 at eachof timings when the first target image is output sequentially at thepreset different locations.

In operation S703 of FIG. 7, the electronic device 100 may determinewhether a gaze of the eyes of the user is directed to the location wherethe first target image is output, based on the obtained gazeinformation.

The electronic device 100 may determine whether a gaze direction of theeyes of the user converges on the location where the first target imageis output at a timing when the first target image is output.

In operation S704 of FIG. 7, according to a result of the determinationof operation S703, the electronic device 100 may determine an impairedarea of an entire visual field based on the location of the output firsttarget image.

While the first target image (e.g., a star-shaped dot) is being outputsequentially at the preset different locations, the electronic device100 may determine a certain area including the location where the firsttarget image is output when the gaze direction of the eyes of the userdoes not converge on the location where the first target image isoutput, as the impaired area.

While the first target image is being output sequentially at the presetdifferent locations, the electronic device 100 may determine a certainarea including the location where the first target image is output whenthe gaze direction of the eyes of the user converges on the locationwhere the first target image is output, as a normal area of vision ofthe user.

In operation S705 of FIG. 7, the electronic device 100 may store avision map based on the determined impaired area.

The electronic device 100 may generate the vision map related to theimpaired area, and store the vision map in the memory 130. Theelectronic device 100 may generate the vision map related to theimpaired area, and the normal area excluding the impaired area from theentire visual field, and store the vision map in the memory 130.

According to an embodiment of the disclosure, the electronic device 100may provide assistance to the user to recognize the impaired area ofvision of the user, based on the previously stored vision map.

FIG. 8 is a view for describing an example in which the electronicdevice 100 diagnoses visual impairment based on obtaining of gazeinformation, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may provide a diagnosis image for allowing a user wearing the electronicdevice 100 to self-diagnose visual impairment of the user. Theelectronic device 100 may control the optical engine 141 to output acertain diagnosis image through the waveguide 142. The electronic device100 may induce the user to look at a target image included in thediagnosis image, and determine whether a gaze of the user is directed toa location where the target image is output. The electronic device 100may determine visual impairment of the user and an impaired area bydetecting a location where the target image is displayed but the usermay not recognize the target image.

Referring to FIG. 8, at a preliminary stage for diagnosing visualimpairment, the electronic device 100 may output a reference targetimage 800 at a preset location of an entire visual field of the user(e.g., the center of the entire visual field).

For example, upon determining that the gaze of the user converges on thereference target image 800, the electronic device 100 may startdiagnosing visual impairment.

The electronic device 100 may provide the diagnosis image including thetarget image sequentially at different locations.

Referring to FIG. 8, the electronic device 100 may output the targetimage (e.g., a star-shaped dot) at a first location 801, and obtain gazeinformation by using the gaze tracking sensor 152. The electronic device100 may output the target image at a second location 802, and obtaingaze information by using the gaze tracking sensor 152. The electronicdevice 100 may output the target image at a third location 803, andobtain gaze information by using the gaze tracking sensor 152.

The electronic device 100 may determine whether a gaze of the eyes ofthe user is directed to a location where the target image is output,based on the gaze information obtained using the gaze tracking sensor152. For example, the electronic device 100 may determine that the gazeof the eyes of the user is directed to the location where the targetimage is output when the target image is displayed at the first andsecond locations 801 and 802, and determine a certain area including thefirst and second locations 801 and 802 as a normal area of vision of theuser.

The electronic device 100 may determine that the gaze of the eyes of theuser is not directed to the location where the target image is outputwhen the target image is displayed at the third location 803, anddetermine a certain area including the third location 803 as an impairedarea of vision of the user. The electronic device 100 may determine thatthe user may not recognize the target image displayed at the thirdlocation 803, and determine the certain area including the thirdlocation 803 as a lost area of vision of the user.

FIG. 9 is a view for describing an example in which the electronicdevice 100 diagnoses visual impairment of the left or right eye,according to an embodiment of the disclosure.

Referring to FIG. 9, at a preliminary stage for diagnosing visualimpairment, the electronic device 100 may output a reference targetimage 900 at a preset location of an entire visual field of a user(e.g., the center of the entire visual field). For example, upondetermining that a gaze of the user converges on the reference targetimage 900, the electronic device 100 may start diagnosing visualimpairment.

According to an embodiment of the disclosure, in order to detect a lostarea of each of the left and right eyes of the user, the electronicdevice 100 may provide the same diagnosis image including a target imagesequentially to the left-eye waveguide 142L (see FIG. 3A) and theright-eye waveguide 142R (see FIG. 3A).

Referring to FIG. 9, the electronic device 100 may output the targetimage (e.g., a star-shaped dot) at a first location 901 on the left-eyewaveguide 142L (see FIG. 3A) (e.g., a top-left side of vision of theuser), and then output the target image (e.g., a star-shaped dot) at asecond location 902 on the right-eye waveguide 142R (see FIG. 3A) (e.g.,a top-left side of vision of the user).

In general, when a user with normal vision opens both eyes, although thetarget image is output only on the left-eye waveguide 142L (see FIG.3A), a gaze of the left eye as well as a gaze of the right eye of theuser are directed to a location of the target image. Likewise, althoughthe target image is output only on the right-eye waveguide 142R (seeFIG. 3A), a gaze of the right eye as well as a gaze of the left eye ofthe user are directed to a location of the target image.

The electronic device 100 may obtain gaze information by using the gazetracking sensor 152 when the target image is output at the firstlocation 901 on the left-eye waveguide 142L (see FIG. 3A) (e.g., thetop-left side of vision of the user). The electronic device 100 mayobtain gaze information by using the gaze tracking sensor 152 when thetarget image is output at the second location 902 on the right-eyewaveguide 142R (see FIG. 3A) (see FIG. 3A) (e.g., the top-left side ofvision of the user).

The electronic device 100 may determine whether a gaze of the eyes ofthe user is directed to a location where the target image is output,based on the gaze information obtained using the gaze tracking sensor152. For example, the electronic device 100 may determine that the gazeof the eyes of the user is directed to the location where the targetimage is output when the target image is displayed at the first andsecond locations 901 and 902 on the left-eye waveguide 142L (see FIG.3A) and the right-eye waveguide 142R (see FIG. 3A), and determine acertain area including the first and second locations 901 and 902 as anormal area for both of the left and right eyes of the user.

FIG. 10 is a view for describing another example in which the electronicdevice 100 diagnoses visual impairment of the left or right eye,according to another embodiment of the disclosure.

Referring to FIG. 10, the electronic device 100 may output a targetimage (e.g., a star-shaped dot) at a third location 903 on the left-eyewaveguide 142L (see FIG. 3A) (e.g., a bottom-right side of vision of auser), and then output the target image (e.g., a star-shaped dot) at afourth location 904 on the right-eye waveguide 142R (see FIG. 3A) (e.g.,a bottom-right side of vision of the user).

The electronic device 100 may obtain gaze information by using the gazetracking sensor 152 when the target image is output at the thirdlocation 903 on the left-eye waveguide 142L (see FIG. 3A) (e.g., thebottom-right side of vision of the user). The electronic device 100 mayobtain gaze information by using the gaze tracking sensor 152 when thetarget image is output at the fourth location 904 on the right-eyewaveguide 142R (see FIG. 3A) (see FIG. 3A) (e.g., the bottom-right sideof vision of the user).

The electronic device 100 may determine whether a gaze of the eyes ofthe user is directed to a location where the target image is output,based on the gaze information obtained using the gaze tracking sensor152.

The electronic device 100 may determine that the user may not recognizethe target image provided on the left-eye waveguide 142L (see FIG. 3A)or the right-eye waveguide 142R (see FIG. 3A), and determine visualimpairment of the left or right eye of the user.

For example, when a user with visual impairment of the right eye opensboth eyes, and when the target image is output only on the left-eyewaveguide 142L (see FIG. 3A), a gaze of the left eye as well as a gazeof the right eye of the user are directed to a location of the targetimage. However, when the target image is output only on the right-eyewaveguide 142R (see FIG. 3A), the user may not recognize the targetimage due to visual impairment of the right eye and thus a gaze of theright eye as well as a gaze of the left eye of the user are not directedto a location of the target image.

Referring to FIG. 10, the electronic device 100 may determine that thegaze of the eyes of the user is directed to the location where thetarget image is output when the target image is output at the thirdlocation 903 on the left-eye waveguide 142L (see FIG. 3A), and determinea certain area including the third location 903 as a normal area for theleft eye of the user.

Meanwhile, the electronic device 100 may determine that the gaze of theeyes of the user is not directed to the location where the target imageis output when the target image is output at the fourth location 904 onthe right-eye waveguide 142R (see FIG. 3A), and determine a certain areaincluding the fourth location 904 as a lost area for the right eye ofthe user.

FIGS. 7 to 10 are merely to describe embodiments of the disclosure, andthe scope of the disclosure is not limited thereto.

FIG. 11 is a flowchart of a method, performed by the electronic device100, of diagnosing visual impairment based on obtaining of a gestureinput, according to an embodiment of the disclosure.

In operation S1101 of FIG. 11, the electronic device 100 may control theoptical engine 141 to output a second target image through the waveguide142.

According to an embodiment of the disclosure, a diagnosis image fordiagnosing visual impairment of a user may include a target imageconfigured as a certain-shaped dot or a certain pattern. For example,the second target image may include a plurality of dots aligned atcertain intervals, an arbitrary closed curve, or a wave pattern, but isnot limited thereto.

In operation S1102 of FIG. 11, the electronic device 100 may outputguidance information related to the second target image, and obtain agesture input of the user by using the depth sensor 153.

According to an embodiment of the disclosure, the electronic device 100may output, through the audio outputter 185, guidance information forinstructing the user wearing the electronic device 100 to trace apattern of the second target image displayed through the waveguide 142with a finger (e.g., ‘Trace the displayed pattern with your finger’).

The electronic device 100 may obtain the gesture input of the user byrecognizing a shape or a motion pattern of a hand of the user, based ondepth information of fingers sensed by the depth sensor 153.

According to an embodiment of the disclosure, the electronic device 100may output, through the audio outputter 185, guidance information forinstructing the user wearing the electronic device 100 to move the headaccording to the pattern of the second target image displayed throughthe waveguide 142 (e.g., ‘Move your head according to the displayedpattern’).

The electronic device 100 may obtain the gesture input of the user basedon a motion pattern of the head of the user sensed using the motionsensor 151.

In operation S1103 of FIG. 11, the electronic device 100 may obtain auser input pattern based on the obtained gesture input of the user. Inoperation S1104 of FIG. 11, the electronic device 100 may compare thesecond target image to the obtained user input pattern.

According to an embodiment of the disclosure, the electronic device 100may determine whether the obtained user input pattern corresponding tothe gesture input of the user matches the pattern of the second targetimage output through the waveguide 142 within a certain range.

In operation S1105 of FIG. 11, the electronic device 100 may determinean impaired area based on a location of the output second target imageaccording to a result of the comparison.

When vision of the eyes of the user includes a distorted area, thecertain pattern of the second target image displayed at a locationcorresponding to the distorted area may appear distorted to the eyes ofthe user and thus the user input pattern may not match the pattern ofthe second target image.

When vision of the eyes of the user includes a lost area, the certainpattern of the second target image displayed at a location correspondingto the lost area may not be seen by the eyes of the user and thus theuser input pattern may not match the pattern of the second target image.

The electronic device 100 may determine the distorted area or the lostarea based on an area where the second target image does not match theuser input pattern.

FIG. 12 is a view for describing an example in which the electronicdevice 100 diagnoses visual impairment based on obtaining of a gestureinput, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may provide a diagnosis image for allowing a user wearing the electronicdevice 100 to self-diagnose visual impairment of the user. Theelectronic device 100 may control the optical engine 141 to output acertain diagnosis image through the waveguide 142. The electronic device100 may induce the user to trace a pattern of a target image included inthe diagnosis image with a finger while looking at the pattern, anddetermine whether a user input pattern corresponding to the gestureinput of the user matches the pattern of the target image. Theelectronic device 100 may determine visual impairment of the user and animpaired area by detecting a location where the user may not accuratelyrecognize the pattern of the displayed target image.

Referring to FIG. 12, at a preliminary stage for diagnosing visualimpairment, the electronic device 100 may output a point of gaze 1202 ata preset location of an entire visual field of the user (e.g., thecenter of the entire visual field). The electronic device 100 may inducethe user to fix the eyes on the point of gaze 1202 during visualimpairment diagnosis based on a gesture input. Upon determining that agaze of the user converges on the location where the point of gaze 1202is output, the electronic device 100 may start diagnosing visualimpairment.

The electronic device 100 may provide a diagnosis image including atarget image 1201 configured as a certain pattern (e.g., an arbitrarydashed closed curve). The electronic device 100 may display a currentlocation guidance marker 1203 at a point of the target image 1201, andinduce the user to trace the pattern of the target image 1201 with afinger in an input proceeding direction from a location of the currentlocation guidance marker 1203.

The electronic device 100 may obtain, using the depth sensor 153, agesture input of the user for tracing the pattern of the target image1201 with a finger. The electronic device 100 may determine whether auser input pattern 1205 corresponding to the obtained gesture inputmatches the pattern of the target image 1201.

Referring to FIG. 12, the electronic device 100 may determine that theuser input pattern 1205 matches the pattern of the target image 1201within a certain range, and determine a certain area including alocation where the target image 1201 is output, as a normal area ofvision of the user.

FIG. 13 is a view for describing another example in which the electronicdevice 100 diagnoses visual impairment based on obtaining of a gestureinput, according to an embodiment of the disclosure.

Referring to FIG. 13, the electronic device 100 may output a point ofgaze 1302 at a preset location of an entire visual field of a user(e.g., the center of the entire visual field), and induce the user tofix the eyes on the point of gaze 1302 during visual impairmentdiagnosis based on a gesture input.

The electronic device 100 may provide a diagnosis image including atarget image 1301 configured as a certain pattern (e.g., an arbitrarydashed closed curve). The electronic device 100 may induce the user totrace the pattern of the target image 1301 with a finger in an inputproceeding direction from a location of a current location guidancemarker 1303.

The electronic device 100 may obtain, using the depth sensor 153, agesture input of the user for tracing the pattern of the target image1301 with a finger. The electronic device 100 may determine whether auser input pattern 1305 corresponding to the obtained gesture inputmatches the pattern of the target image 1301.

Referring to FIG. 13, the electronic device 100 may determine that theuser input pattern 1305 does not match the pattern of the target image1301 within a certain range, and determine a certain area including theunmatched area as an impaired area of vision of the user.

The electronic device 100 may determine an area 1307 where the userinput pattern 1305 does not match the pattern of the target image 1301by more than a certain distance, as a distorted area.

The electronic device 100 may determine an area 1306 where the gestureinput of the user corresponding to the pattern of the target image 1301is not obtained, as a lost area.

FIG. 14 is a view for describing an example in which the electronicdevice 100 detects a distorted area, according to an embodiment of thedisclosure.

Referring to FIG. 14, the electronic device 100 may output a point ofgaze 1402 at a preset location of an entire visual field of a user(e.g., the center of the entire visual field), and induce the user tofix the eyes on the point of gaze 1402 during visual impairmentdiagnosis based on a gesture input.

The electronic device 100 may provide a diagnosis image including atarget image 1401 configured as a certain pattern (e.g., a pattern ofalignment of a plurality of dots).

The electronic device 100 may induce the user to trace the pattern ofthe target image 1401 (e.g., the pattern of alignment of the pluralityof dots) with a finger. Alternatively, the electronic device 100 mayinduce a gesture input for pointing a finger at the plurality of dotsone by one according to the pattern of alignment of the plurality ofdots.

Referring to FIG. 14, for example, the target image 1401 in which aplurality of dots are aligned in a line and a dot is misaligned by about2 degrees may be output through the left-eye waveguide 142L (see FIG.3A).

The electronic device 100 may determine whether a user input patterncorresponding to the gesture input obtained using the depth sensor 153matches the pattern of the target image 1401.

The electronic device 100 may determine that the user input pattern 1405matches the pattern of the target image 1401 within a certain range, anddetermine a certain area including a location where the target image1401 is displayed, as a normal area of vision of the user.

Alternatively, when a user input for aligning the misaligned dot amongthe dots included in the target image 1401 with the other adjacent dotsin a line is obtained, the electronic device 100 may determine a certainarea including the location where the target image 1401 is displayed, asa normal area of vision of the user.

FIG. 15 is a view for describing another example in which the electronicdevice 100 detects a distorted area, according to an embodiment of thedisclosure.

According to an embodiment of the disclosure, the electronic device 100may determine whether an area, which is determined not to be a lost areaof vision of a user based on obtaining of gaze information, is adistorted area.

According to an embodiment of the disclosure, the distorted area refersto an area which is visible but appears distorted to the eyes of theuser. Therefore, although the user may recognize one dot when the dot isdisplayed on the distorted area of vision of the user, the use may notaccurately recognize a pattern of alignment of a plurality of dotsaligned and displayed on the distorted area.

According to an embodiment of the disclosure, in order to diagnose alost area of vision of the user based on obtaining of gaze information,the electronic device 100 may induce the user to look at a target imageincluded in a diagnosis image, and determine whether a gaze of the useris directed to a location where the target image is output.

Referring to FIG. 15, the electronic device 100 may output a firsttarget image 1501 (e.g., a rectangular dot) at a bottom-left side ofvision of the user, and obtain gaze information by using the gazetracking sensor 152.

The electronic device 100 may determine whether a gaze of the eyes ofthe user is directed to the location where the first target image 1501is output, based on the gaze information obtained using the gazetracking sensor 152. For example, upon determining that the gaze of theeyes of the user is directed to the location where the first targetimage 1501 is output when the first target image 1501 is displayed atthe bottom-left side of vision of the user, the electronic device 100may determine that a certain area including the bottom-left side ofvision of the user is not a lost area.

Referring to FIG. 15, in order to diagnose a distorted area of vision ofthe user based on obtaining of a gesture input, the electronic device100 may output a second target image 1502 configured as a pattern ofalignment of a plurality of dots, at the same location where the firsttarget image 1501 is output.

The electronic device 100 may output a point of gaze 1503 at a presetlocation of an entire visual field of the user (e.g., the center of theentire visual field), and induce the user to fix the eyes on the pointof gaze 1503 during visual impairment diagnosis based on a gestureinput.

The electronic device 100 may induce the user to trace the pattern ofthe second target image 1502 with a finger. Alternatively, theelectronic device 100 may induce a gesture input for pointing a fingerat the plurality of dots one by one according to the pattern ofalignment of the plurality of dots.

The electronic device 100 may determine that a user input patterncorresponding to the gesture input obtained using the depth sensor 153does not match the pattern of the second target image 1502 within acertain range, and determine a certain area including the location wherethe second target image 1502 is displayed, as a distorted area of visionof the user.

The electronic device 100 may determine that the user may not accuratelyrecognize the pattern of alignment of the second target image 1502output through the left-eye waveguide 142L (see FIG. 3A), and determinea certain area including the location where the second target image 1502is displayed on an entire visual field of the left eye of the user, as adistorted area.

FIG. 16 is a view for describing another example in which the electronicdevice 100 detects an impaired area of vision of a user, according to anembodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may provide a diagnosis image by gradually reducing a radius of a targetimage configured as a circular closed curve.

The electronic device 100 may provide a diagnosis image by graduallyreducing the size of an arbitrary circular closed curve including a wavepattern from the periphery to the center of vision of a user, and thusperform more precise diagnosis by minimizing a non-diagnosed area in anentire visual field of the user.

According to an embodiment of the disclosure, the electronic device 100may induce the user wearing the electronic device 100 to trace thepattern of the target image with a finger. The electronic device 100 mayobtain a gesture input of the user for tracing the pattern of the targetimage with a finger, by using the depth sensor 153.

Referring to FIG. 16, the electronic device 100 may determine whether afirst user input pattern 1602 corresponding to an obtained gesture inputmatches a pattern of a first target image 1601. The electronic device100 may determine whether a second user input pattern 1604 correspondingto an obtained gesture input matches a pattern of a second target image1603. The electronic device 100 may determine whether a third user inputpattern 1606 corresponding to an obtained gesture input matches apattern of a third target image 1605.

According to an embodiment of the disclosure, the electronic device 100may determine an area 1607 where the gesture input of the usercorresponding to the pattern of the first target image 1601 is notobtained, as a lost area.

The electronic device 100 may determine an area 1608 where the firstuser input pattern 1602 does not match the pattern of the first targetimage 1601 by more than a certain distance, as a distorted area. Theelectronic device 100 may determine an area 1609 where the second userinput pattern 1604 does not match the pattern of the second target image1603 by more than a certain distance, as a distorted area.

The electronic device 100 may determine an area excluding the distortedand lost areas from the entire visual field, as a normal area of visionof the user.

The electronic device 100 may determine that the third user inputpattern 1606 matches the pattern of the third target image 1605 within acertain range, and determine a certain area including a location wherethe third target image 1605 is output, as a normal area of vision of theuser.

According to an embodiment of the disclosure, the electronic device 100may generate a vision map related to the normal area of vision of theuser, and an impaired area including the distorted and lost areas, andstore the vision map in the memory 130.

According to an embodiment of the disclosure, the electronic device 100may provide assistance to the user to recognize the impaired area ofvision of the user, based on the vision map.

FIGS. 11 to 16 are merely to describe embodiments of the disclosure, andthe scope of the disclosure is not limited thereto.

FIG. 17 is a flowchart of a method, performed by the electronic device100, of calculating a degree of distortion, according to an embodimentof the disclosure.

According to an embodiment of the disclosure, when vision of a user withvisual impairment includes a distorted area, a degree of distortion mayvary. The electronic device 100 may diagnose the degree of distortionthrough visual impairment diagnosis based on obtaining of a gestureinput.

In operation S1701 of FIG. 17, the electronic device 100 may determinean area where a target image does not match a user input pattern. Inoperation S1702 of FIG. 17, the electronic device 100 may determine animpaired area based on the area where the target image does not matchthe user input pattern.

According to an embodiment of the disclosure, the electronic device 100may induce the user to trace a pattern of the target image included in adiagnosis image with a finger while looking at the pattern, anddetermine whether the user input pattern corresponding to the gestureinput of the user matches the pattern of the target image. Theelectronic device 100 may determine the impaired area of vision of theuser by detecting a location where the user may not accurately recognizethe pattern of the displayed target image.

In operation S1703 of FIG. 17, the electronic device 100 may determine adegree of distortion in the impaired area based on a distance betweenthe target image and the user input pattern.

According to an embodiment of the disclosure, the electronic device 100may calculate the distance between the target image and the user inputpattern in the area where the target image does not match the user inputpattern. The electronic device 100 may determine the degree ofdistortion in a distorted area based on the calculated distance. Theelectronic device 100 may determine a higher degree of distortion for alonger distance.

FIG. 18 is a view for describing an example in which the electronicdevice 100 calculates a degree of distortion, according to an embodimentof the disclosure.

According to an embodiment of the disclosure, the electronic device 100may determine a degree of distortion in a distorted area of vision of auser by calculating a distance between a target image and a user inputpattern corresponding to a gesture input of the user.

Referring to FIG. 18, for example, the electronic device 100 maycalculate a first distance 1804 between a first user input pattern 1803and a first target image 1802 in a first distorted area 1801. Forexample, the electronic device 100 may calculate a second distance 1808between a second user input pattern 1807 and a second target image 1806in a second distorted area 1805.

According to an embodiment of the disclosure, the electronic device 100may determine a higher degree of distortion for a longer distance, anddetermine a lower degree of distortion for a shorter distance.

According to an embodiment of the disclosure, the electronic device 100may quantify the degree of distortion to, for example, score 30 or score65, according to a predetermined criterion.

FIGS. 17 and 18 are merely to describe embodiments of the disclosure,and the scope of the disclosure is not limited thereto.

FIG. 19 is a flowchart of a method, performed by the electronic device100, of outputting a guidance image for notifying an impaired area ofvision of a user, according to an embodiment of the disclosure.

In operation S1901 of FIG. 19, the electronic device 100 may store avision map.

According to an embodiment of the disclosure, the electronic device 100may generate the vision map including information about an impairedarea, by diagnosing visual impairment of a user wearing the electronicdevice 100, and store the vision map in the memory 130.

In operation S1902 of FIG. 19, the electronic device 100 may control theoptical engine 141 to output a guidance image for notifying the user ofthe impaired area through the waveguide 142, based on the vision map.

According to an embodiment of the disclosure, the guidance image mayinclude, for example, a solid or dashed outline indicating the boundaryof the impaired area, but is not limited thereto.

FIG. 20 is a view for describing an example in which the electronicdevice 100 outputs a guidance image for notifying an impaired area,according to an embodiment of the disclosure.

Referring to FIG. 20, the electronic device 100 may output a guidanceimage 2002 indicating a location of an impaired area 2001 of vision of auser wearing the electronic device 100, in the form of an outline of theimpaired area 2001 on a normal area outside the impaired area 2001 in areal scene seen by the user through the waveguide 142.

According to an embodiment of the disclosure, when the user wearing theelectronic device 100 sees the real scene, the user may recognize thatthe real scene seen by the user includes the impaired area of vision ofthe user, based on the guidance image.

FIG. 21 is a view for describing an example in which the electronicdevice 100 outputs a guidance image on a display, according to anembodiment of the disclosure. The scene of FIG. 21 is that of a persondescending a staircase.

Referring to FIG. 21, the electronic device 100 may identify a distortedarea 2101 and a lost area 2102 in an entire visual field of a userwearing the electronic device 100, based on a previously stored visionmap.

According to an embodiment of the disclosure, the entire visual field ofthe eyes of the user may include an entire display area on the waveguide142 of the electronic device 100.

The electronic device 100 may determine a location of a distorted area2103 on the waveguide 142 corresponding to the distorted area 2101, anddetermine a location of a lost area 2104 on the waveguide 142corresponding to the lost area 2102, on the entire display area of thewaveguide 142.

The electronic device 100 may control the optical engine 141 to displaya first guidance image 2105 indicating an outline of the distorted area2103 and a second guidance image 2106 indicating an outline of the lostarea 2104, through the waveguide 142.

According to an embodiment of the disclosure, when the user sees a realscene through the waveguide 142, the user may recognize that a realworld object of the real scene, which is located in a direction and at adistance where a guidance image is displayed, is not accurately visibledue to visual impairment of the user. In FIG. 21, at the place and timeillustrated a portion of a step of the staircase is a real world objectin the field of view of the outline (second guidance image 2106) of thelost area 2104.

FIGS. 19 to 21 are merely to describe embodiments of the disclosure, andthe scope of the disclosure is not limited thereto.

FIG. 22 is a flowchart of a method, performed by the electronic device100, of outputting an object based on a prism mode, according to anembodiment of the disclosure.

In operation S2201 of FIG. 22, the electronic device 100 may store avision map.

According to an embodiment of the disclosure, the electronic device 100may generate the vision map including information about an impairedarea, by diagnosing visual impairment of a user wearing the electronicdevice 100, and store the vision map in the memory 130.

In operation S2202 of FIG. 22, the electronic device 100 may detect anobject included in the impaired area, based on the vision map.

According to an embodiment of the disclosure, the electronic device 100may identify the impaired area including distorted and lost areas in anentire visual field of the user wearing the electronic device 100, basedon the previously stored vision map.

According to an embodiment of the disclosure, the electronic device 100may determine whether a real world object is present in the impairedarea of vision of the user.

The electronic device 100 may obtain information indicating whether thereal world object is present in front of the electronic device 100, adirection and distance of the real world object, etc., based on depthinformation of the real world object sensed by the depth sensor 153.

In operation S2203 of FIG. 22, the electronic device 100 may apply aprism mode to at least a partial area of the varifocal lens 145corresponding to the impaired area. The electronic device 100 may applythe prism mode to at least the partial area of the varifocal lens 145 insuch a manner that an image of the object detected in operation S2202 isrefracted and moved to a normal area of vision of the user.

The prism mode according to an embodiment of the disclosure is a mode inwhich refractive power of a specific area of the varifocal lens 145 ischanged. The refractive power of the varifocal lens 145 may be changedby changing orientation of LC molecules in the varifocal lens 145.

The electronic device 100 may exert control to change a path of lightpassing through the specific area of the varifocal lens 145corresponding to the impaired area, by applying the prism mode to thespecific area of the varifocal lens 145 corresponding to the impairedarea of vision of the user.

When the prism mode is applied to the specific area of the varifocallens 145 corresponding to the impaired area, a voltage may be applied tohave a phase profile corresponding to the prism mode, and thus therefractive power may be changed. As such, the path of light passingthrough the specific area of the varifocal lens 145 corresponding to theimpaired area may be changed, and thus the image of the object includedin the impaired area may be refracted and output on the normal area ofthe entire visual field.

In operation S2204 of FIG. 22, the electronic device 100 may control theoptical engine 141 to display a virtual image indicating that the imageof the object is moved to the normal area of the entire visual field,based on application of the prism mode.

When the image of the object detected in the impaired area is refractedby the varifocal lens 145 to which the prism mode is applied, and ismoved to the normal area, the electronic device 100 may control theoptical engine 141 to output a virtual image, e.g., a mark, an icon, ora certain image, for notifying the user that the image is moved to thenormal area so as to be visible to the user.

FIG. 23 is a view for describing application of a prism mode, accordingto an embodiment of the disclosure.

According to an embodiment of the disclosure, a prism mode may beapplied to at least a partial area of the varifocal lens 145corresponding to an impaired area of vision of a user.

When the prism mode is applied to a specific area of the varifocal lens145 corresponding to the impaired area of vision of the user, a path oflight passing through the specific area may be changed and thus an imageof an object included in the impaired area may be refracted and moved toa normal area of vision of the user.

For example, compared to a path 2301 of light passing through thevarifocal lens 145 when a lens mode is applied, when the prism mode isapplied to the specific area of the varifocal lens 145, light passingthrough the specific area may be refracted to a path 2302 of light. Whenthe prism mode is applied, orientation of LC molecules may be changed byapplying a voltage to the specific area of the varifocal lens 145 tohave the phase profile 632 (see FIG. 6) corresponding to the prism mode,and thus refractive power may be changed.

As such, an image of a real world object displayed at a first location2303 in the specific area of the varifocal lens 145 when the lens modeis applied may be moved to a second location 2304 in the normal area ofvision of the user due to a change of a path of light passing throughthe specific area when the prism mode is applied.

FIG. 24 is a view for describing an example in which the electronicdevice 100 outputs an object based on a prism mode, according to anembodiment of the disclosure.

Referring to FIG. 24, when a user wearing the electronic device 100 seesa real scene, the user may not recognize a real world object 2401included in an impaired area 2402 of vision of the user.

According to an embodiment of the disclosure, the electronic device 100may exert control to move an image of the real world object 2401included in the impaired area 2402 to a partial area in a normal area ofvision of the user, by applying a prism mode to a specific area of thevarifocal lens 145 corresponding to the impaired area 2402 of vision ofthe user.

As such, when the user wearing the electronic device 100 sees the realscene, the user may recognize the real world object 2401, which isincluded in the impaired area 2402 of vision of the user and thus is notvisible to the user, at another location as a moved real world objectimage 2403.

According to an embodiment of the disclosure, the electronic device 100may output a virtual certain icon or image 2404 around the moved realworld object image 2403 in such a manner that the user may recognizethat the real world object image 2403 visible to the user in the normalarea of vision of the user is moved from the impaired area of vision ofthe user.

FIGS. 22 to 24 are merely to describe embodiments of the disclosure, andthe scope of the disclosure is not limited thereto.

Meanwhile, the afore-described embodiments of the disclosure may bewritten as computer-executable programs, and be implemented by ageneral-purpose digital computer for operating the programs by using acomputer-readable medium. Data structures used in the afore-describedembodiments of the disclosure may be recorded on the computer-readablemedium via various means. The afore-described embodiments of thedisclosure may be implemented in the form of a recording mediumincluding computer-executable instructions, e.g., computer-executableprogram modules. For example, methods implemented as software modules oralgorithms may be stored in a computer-readable recording medium ascomputer-readable and—executable codes or program commands.

The computer-readable medium may be an arbitrary recording mediumaccessible by a computer, and include volatile, non-volatile,detachable, and non-detachable media. Examples of the computer-readablemedium include magnetic storage media (e.g., read-only memory (ROM),floppy disks, and hard disks) and optical recording media (e.g., compactdisc-ROM (CD-ROM) and digital versatile discs (DVDs)), but are notlimited thereto. The computer-readable medium may include a computerstorage medium and a communication medium.

A plurality of computer-readable recording media may be distributed overnetwork-coupled computer systems, and data, e.g., program instructionsand codes, stored in the distributed recording media may be executed byat least one computer.

Particular implementations described herein merely correspond toembodiments of the disclosure and do not limit the scope of thedisclosure in any way. For brevity, descriptions of known electronicconfigurations, control systems, software, and other functional aspectsof the systems may not be provided herein.

While the disclosure has been particularly shown and described withreference to embodiments of the disclosure, it will be understood by oneof ordinary skill in the art that various changes in form and detailsmay be made therein without departing from the scope of the disclosure.Therefore, it should be understood that the afore-described embodimentsof the disclosure are illustrative in all aspects and do not limit thedisclosure. For example, each element described as a single element maybe implemented in a distributed manner and, likewise, elements describedas distributed elements may be implemented in a combined manner.

All examples and illustrative terms, e.g., “etc.”, used herein aremerely to describe the disclosure in detail and the scope of thedisclosure is not limited by those examples and illustrative termsunless defined in the claims.

Moreover, no element is essential for implementation of the disclosureunless the element is particularly described as being “essential” or“critical”.

It will be understood by one of ordinary skill in the art that theembodiments of the disclosure may be modified without departing from thescope of the disclosure.

It should be understood that various changes in form and details may bemade in the embodiments of the disclosure and that the embodiments ofthe disclosure cover all modifications, equivalents, and alternativesfalling within the scope of the disclosure. Therefore, theafore-described embodiments of the disclosure should be considered in adescriptive sense only and not for purposes of limitation.

The scope of the disclosure is defined not by the detailed descriptionof the disclosure but by the appended claims, and all variations derivedfrom the scope defined by the claims and their equivalents will beconstrued as being included in the scope of the disclosure.

As used herein, the term “. . . unit” or “module” denotes an entity forperforming at least one function or operation, and may be implemented ashardware, software, or a combination of hardware and software.

The “unit” or “module” may also be implemented by a program stored in anaddressable storage medium and executable by a processor.

For example, the term “unit” or “module” may be implemented by elements(e.g., software elements, object-oriented software elements, classelements, and task elements), processes, functions, attributes,procedures, subroutines, segments of program code, drivers, firmware,micro-codes, circuits, data, a database, data structures, tables,arrays, or variables.

As used herein, the expression “A may include one of a1, a2, and a3”broadly means that an example of an element that may be included inelement A is a1, a2, or a3.

The expression does not limit the element that may be included inelement A, to a1, a2, or a3. Therefore, it should be noted that theexpression is not restrictively construed to exclude elements other thana1, a2, and a3, from examples of the element that may be included in A.

The expression means that A may include a1, include a2, or include a3.The expression does not mean that elements included in A are alwaysselectively determined within a certain set. For example, it should benoted that the expression is not restrictively construed to limit theelement included in element A, to a1, a2, or a3 selected from a setincluding a1, a2, and a3.

1. An electronic device comprising: a display comprising an opticalengine and a waveguide; a gaze tracking sensor; a memory configured tostore one or more instructions; and a processor configured to executethe one or more instructions to: control the optical engine to output afirst target image through the waveguide at preset different locations;obtain gaze information of eyes of a user corresponding to a location ofthe first target image by using the gaze tracking sensor; determinewhether a gaze of the eyes of the user is directed to the location wherethe first target image is output, based on the gaze information;according to a first result of the determination, determine an impairedarea of an entire visual field; and store a vision map based on theimpaired area.
 2. The electronic device of claim 1, further comprising adepth sensor, wherein the processor is further configured to execute theone or more instructions to: control the optical engine to output asecond target image through the waveguide; output guidance informationrelated to the second target image; obtain, by using the depth sensor, agesture input of the user associated with the guidance information;obtain a user input pattern based on the gesture input of the user;compare the second target image to the user input pattern; and determinethe impaired area based on a second result of the comparison.
 3. Theelectronic device of claim 2, wherein the processor is furtherconfigured to execute the one or more instructions to: determine an areawhere the second target image does not match the user input pattern;determine the impaired area based on the area where the second targetimage does not match the user input pattern; and determine a degree ofdistortion in the impaired area based on a distance between the secondtarget image and the user input pattern.
 4. The electronic device ofclaim 1, wherein the processor is further configured to execute the oneor more instructions to control the optical engine to output a guidanceimage for notifying the user of the impaired area through the waveguide,based on the vision map.
 5. The electronic device of claim 1, furthercomprising: a depth sensor; and a varifocal lens, wherein the processoris further configured to execute the one or more instructions to: detectan object comprised in the impaired area, based on the vision map byusing the depth sensor; apply a prism mode to at least a partial area ofthe varifocal lens corresponding to the impaired area in such a mannerthat an image of the object is refracted and moved to a normal area; andcontrol the optical engine to display a virtual image indicating thatthe image of the object is moved to the normal area of the entire visualfield.
 6. The electronic device of claim 5, wherein the prism mode is amode in which refractive power of at least the partial area of thevarifocal lens is changed, and wherein the refractive power of thevarifocal lens is changed by changing orientation of liquid crystal (LC)molecules in the varifocal lens.
 7. The electronic device of claim 1,wherein the entire visual field comprises an entire display area on thewaveguide, and wherein the preset different locations comprise at leastone of a plurality of locations spaced apart from each other at certainintervals on the waveguide.
 8. The electronic device of claim 1, whereinthe impaired area comprises at least one of a lost area or a distortedarea.
 9. A method of operating an electronic device, the methodcomprising: controlling an optical engine to output a first target imagethrough a waveguide at preset different locations; obtaining gazeinformation of eyes of a user corresponding to a location of the firsttarget image by using a gaze tracking sensor when the first target imageis output; determining, based on the gaze information, whether a gaze ofthe eyes of the user is directed to the first target image; according toa first result of the determination, determining an impaired area of anentire visual field; and storing a vision map based on the impairedarea.
 10. The method of claim 9, comprising: controlling the opticalengine to output a second target image through the waveguide; outputtingguidance information related to the second target image; obtaining, byusing a depth sensor, a gesture input of the user associated with theguidance information; obtaining a user input pattern based on thegesture input of the user; comparing the second target image to the userinput pattern; and determining the impaired area based on a secondresult of the comparison.
 11. The method of claim 10, wherein thedetermining of the impaired area comprises: determining an area wherethe second target image does not match the user input pattern;determining the impaired area based on the area where the second targetimage does not match the user input pattern; and determining a degree ofdistortion in the impaired area based on a distance between the secondtarget image and the user input pattern.
 12. The method of claim 9,further comprising controlling the optical engine to output a guidanceimage for notifying the user of the impaired area through the waveguide,based on the vision map.
 13. The method of claim 9, comprising:detecting an object comprised in the impaired area, based on the visionmap by using a depth sensor; applying a prism mode to at least a partialarea of a varifocal lens corresponding to the impaired area in such amanner that an image of the object is refracted and moved to a normalarea; and controlling the optical engine to display a virtual imageindicating that the image of the object is moved to the normal area ofthe entire visual field.
 14. The method of claim 13, wherein the prismmode is a mode in which refractive power of at least the partial area ofthe varifocal lens is changed, and wherein the refractive power of thevarifocal lens is changed by changing orientation of liquid crystal (LC)molecules in the varifocal lens.
 15. A computer-readable recordingmedium having recorded thereon a computer program for executing themethod of claim 9.